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APPLICATION  OF  STATISTICAL  TECHNIQUES  TO  MODEL  SENSITIVITY  TESTING 

SUMMARY 


1 • Facts.  - An  Improved  method  of  performing  sensitivity  analy- 
ses on  large  scale  computer  models  was  sought  by  the  US  Amy 
Concepts  Analysis  Agency  (CAA),  8120  Woodmont  Avenue,  Bethesda. 

MO  20014.  The  Methodology  and  Resources  Directorate,  CAA,  initi- 
ated the  task  in  FY  74,  of  developing,  illustrating  and  document- 
ing the  needed  improved  method  on  a time  available  basis.  The 
completion  date  of  the  task  was  September  1974. 

2.  Purpose.  - The  purpose  of  the  task  was  to  provide  a compre- 
hensive method  of  performing  sensitivity  analyses  on  large  scale 
computer  models.  A method  having  greater  efficiency  than  existing 
models  was  desired. 

3.  Objective.  - A means  of  obtaining  more  Information  about  how 
changes  in  input  variables  affect  output  variables  in  a large  scale 
computer  model  was  a prime  objective.  Minimization  of  the  number 
of  computer  runs  without  loss  of  essential  information  was  inherent 
In  the  task. 

4.  Discussion.  - More  and  better  infomation  can  be  obtained 
by  using  factorial  design  of  experiments,  a statistical  technique. 

A factorial  design  is  a plan  in  which  each  factor  under  study  is 
tested  with  each  of  the  other  factors  at  each  level  of  interest. 

This  method  allows  Information  to  be  obtained  on  the  interaction 
effect  of  the  factors  under  study.  The  interaction  information  Is 
important  because  valid  inferences  cannot  be  made  about  any  factor, 
independently  of  other  factors  if  interaction  exists.  That  Is, 

each  factor  effect  is  dependent  upon  the  level  of  the  other  factor(s). 
Frequently,  because  of  the  large  msnber  of  factors  and  levels,  a 
full  (all  factor  level  combinations)  factorial  design  exceeds  the 
lumber  of  computer  runs  available.  The  technique  presented  in  this 
paper,  for  application  of  fractional  factorial  designs,  reduces  the 
number  of  computer  runs  required  without  sacrificing  Interaction 
information.  The  use  and  application  of  fractional  factorial  de- 
signs are  discussed.  An  additional  technique  whereby  time  and 
effort  in  preparation  of  the  inputs  for  the  computer  runs  can  be 
saved  by  chewsing  the  run  order  is  also  discussed.  While  all  tech- 
niques discussed  above  are  widely  used  by  statisticians  and  well 
documented  in  the  statistical  literature,  the  use  of  these  tech- 
niques in  model  sensitivity  testing  is  not  widely  used.  Examples 
of  output  analyses  and  presentation  of  the  results  from  a factorial 
designed  experiment  are  presented  to  demonstrate  how  the  method 
works. 
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5-  Observations*  - The  application  of  fractional  factorial  de- 
slgns  provides  a needed  technique  for  reduction  of  computer  runs 
tflthout  sacrificing  nodel  Interaction  inforaatiou.  bploynent  of 
an  additional  technique  saves  tine  and  effort  In  preparation  of 
Inputs  for  conputer  runs.  This  latter  technique,  vinialzes  the 
factor  level  changes  required  in  performing  a factorial  design 
experiment. 
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application  of  statistical  techniques  to  model  sensitivity  testing 


CHAPTER  I 
INTRODUCTION 


1 • background.  - The  use  of  large  scale  computer  models  for 
guidance  In  real  world  decisions  is  now  coRvnonplace  in  the  Department 
of  Defense,  other  government  agencies,  and  private  Industry.  These 
models  were  developed  from  observations,  data  and  theories  to  predict 
outcomes  or  results  for  conditions  or  situations  that  cannot  be 
observed  or  in  cases  where  costs  of  observation  are  excessive. 

These  models  can  be  broadly  categorized  as  stochastic  or  determin- 
istic. Stochastic  or  probablistic  models  attenpt  to  capture  the 
chance  element  associated  with  the  processes  being  modeled,  by 
using  probability  distributions  for  many  of  the  input  variables. 
Random  samples  are  then  drawn  from  the  distributions  mid  manipulated 
to  produce  model  results.  In  deterministic  models,  the  means  or 
expected  values  of  the  variables  are  input  as  user-defined 
constants.  These  input  variables  are  then  used  in  a sequence  of 
analytical  equations  or  expressions  to  produce  model  results.  In 
both  stochastic  and  deterministic  models,  the  question  arises  as  to 
how  sensitive  the  output  factors  are  to  changes  in  the  input  factors. 
Since  these  models  are  comprised  of  many  factors  with  many  inter- 
actions, it  is  very  difficult.  If  not  impossible,  to  analytically 
determine  what  effect  a Change  In  an  input  factor  will  have  on  an 
output  factor.  Therefore,  the  model  is  exercised  with  various 
changes  to  the  input  factors.  This  process  is  generally  referred  to 
as  sensitivity  analysis. 

2.  Purpose.  - Design  of  experiment  techniques  have  been  widely 
used  in  the  statistical  community  mid  is  well  documented  in  the 
literature.  However,  the  use  of  these  techniques  for  model  sensi- 
tivity testing  has  not  been  common.  A description  of  how  more  and 
better  information  can  be  obtained  from  sensitivity  analysis  by 
using  factorial  design  of  experiments  is  given  In  this  paper. 

An  explanation  is  given  of  how  time  and  effort  can  be  saved  in 
preparing  Inputs  by  choosing  the  proper  sequence  in  which  to  make 
the  model  runs.  Some  ways  in  which  the  results  can  be  analyzed 
and  presented  are  also  given. 
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APPLICATION  OF  STATISTICAL  TECHNIQUES  TO  MODEL  SENSITIVITY  TESTING 


CHAPTER  II 

FACTORIAL  DESIGN  VERSUS  ONE-FACTOR-AT -A-TIHE 


1.  One- Factor- At- A-T 1 me . - The  old  traditional  nethod  of  doing 
a sensitivity  analysis  has  been  to  set  all  the  variables  or  factors 
except  one  at  sene  nominal  value  and  then  vary  one  factor.  A value 
higher  or  greater  than  the  nominal  value  is  Irput  for  one  factor 
and  the  model  Is  executed  to  determine  the  effect  on  the  output 
varlable(s).  Then  a value  lower  than  the  nominal  value  Is  Input 
for  the  one  factor  and  the  model  is  rerun.  Sometimes  many  different 
changes  are  made  to  the  Input  variable  and  other  times  only  two 
values,  a nigh  value  and  a low  value,  will  be  used.  Frequently, 
many  different  variables  will  be  tested,  one  at  a time,  each  time 
with  the  remaining  variables  held  fixed  at  a constant  nornlnaT  value. 
Figure  II-l  Illustrates  a One- Factor- A t-A-T i m sensitivity  layout* 
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FIGURE  II-l , Ohe-Factor-At-A-T ine 

where  four  levels  (values)  of  factor  A are  tested  at  only  one  level 
(value)  of  factor  B.  While  this  One- Factor-At-A-T ime  method  does 
provide  some  information  on  how  the  outout  varies  for  a change  In 
the  Input  factor  or  variable,  the  information  is  of  a limited 
nature.  First  of  all,  all  of  the  information  Jbtalned  for  factor 
A Is  for  only  one  level  of  factor  B.  Information  about  the  12 
vacant  cells  is  not  obtained.  Thus,  knowledge  of  how  factor  A 
would  have  affected  the  output  If  factor  8 would  have  been  at  some 
other  level  Is  not  determined.  Secondly,  using  the  One-Factor-At- 
A-Tirae  method,  information  concerning  the  effect  of  the  Interaction 
of  factor  A with  factor  B Is  not  ascertainable.  When  there  Is  an 
interaction  effect  between  two  factors  the  outcome  Is  dependent 
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upon  which  level  each  of  the  factors  Is  set*  l.e.,  one  factor 
behaves  differently  under  different  levels  of  the  other  factor. 

In  figure  1 1-2,  for  exa^»le*  when  factor  B Is  set  at  the  low  level* 
the  output  variable  decreases  when  factor  A is  changed  from  the  low 
level  to  the  high  level.  However*  when  factor  B Is  set  at  the  high 
level*  the  output  variable  Increases  when  factor  A is  changed  fron 
the  low  level  to  the  high  level.  (The  lines  In  Figure  II-Z  are  Merely 
for  connecting  associated  points.)  Thus . factor  A interacts  with 
factor  8. 


FIGURE  II-2*  Interaction  Example 

This  Interaction  information  is  important  since  it  tells  the  model 
user  that  neither  factor  A nor  factor  8 can  be  examined  in  Isola- 
tion due  to  the  fact  that  the  reaction  of  the  output  variable  Is 
dependent  upon  which  level  each  of  these  factors  was  set.  There” 
fore.  It  Is  desirable  to  erplpy  a method  which  permits  the 
assessment  of  at  least  the  two-factor  Interactions. 

2.  Factorial  Design  Method 

a.  The  proposed  method  1$  to  use  principles  of  experimental 
design  ditch  allow  each  variable  (factor)  undeer  study  to  be  examined 
with  each  of  the  other  variables  at  each  level  of  Interest.  Zn 
experimental  design,  this  Is  referred  to  as  a factorial  design.  For 
an  illustrative  example,  consider  three  factors*  A*  8.  and  C*  at  two 
levels  each*  a low  level  (L)  and  a high  level  (H)  (See  Figure  II-3). 
This  example  constitutes  a 23  factorial  experimental  design. 
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FIGURE  11-3,  A 2 Factorial  Experimental  Design 


All  factors*  other  than  A*  8,  and  C,  are  held  at  a constant  nominal 
value.  The  A,  8*  and  C high/low  values  are  chosen  such  that  most 
of  the  values  (90  or  98  percent)  anticipated  for  that  factor  will 
be  between  these  high/low  values.  By  choosing  high/ low  values  near 
the  extremes  of  the  value  spectrum,  the  probability  of  obtaining  a 
good  estimate  of  the  trend  in  the  effects  will  be  increased.  If 
the  high/low  values  chosen  are  too  close  to  each  other,  the  proba- 
bility of  obtaining  false  trends  or  "noise"  is  increased. 

b Next  "expand"  the  23  factorial  design  into  a 33  factorial 
design  and  call  tha  third  level  the  nominal  level.  This  action  will 
illustrate  where  the  traditional  method  runs  would  fit  in  with  those 
of  the  2r  factorial  design  (See  Figure  If-4).  The  quotation  marts 
indicate  the  entries  trfilch  belong  to  the  traditional  method.  From 
this  it  can  be  seen  how  the  traditional  method  runs  are  all 
clustered  about  the  center  of  the  design.  The  runs  required  for 
each  method,  in  addition  to  the  base  case,  are  listed  in  Figure 
II-5  for  ease  of  comparison.  Note  the  factorial  method  requires 
two  more  runs  than  the  traditional  method.  Next  observe  the  output 
effects  that  can  be  assessed  for  each  of  the  methods  (See  Figure 
1 1-6).  Here  it  can  be  seen  that  for  two  additional  nets,  using  the 
factorial  method,  information  is  obtained  on  the  output  effect  of 
the  AB,  AC,  8C,  and  ABC  interactions  in  addition  to  the  main  factors 
(A,  B,  and  C).  In  addition  to  being  able  to  assess  the  interaction 
effects  using  the  factorial  nethod.  more  Information  is  obtained 


Factor  A 


on  the  factors  at  the  points  of  west  Interest.  Exalte:  Observe 
factor  A for  sane  Measure  of  Effectiveness  (NOE)  for  each  eethod 
(See  Figure  1 1-7). 
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FIGURE  II«4.  ‘Expanded*  Factorial  Design 
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FIGURE  I I -7,  Evaluation  Points 


Traditional  and  Factorial  Methods 


The  masters  in  parenthesis  are  the  nuriier  of  values  used  to  estimate 
each  point.  For  the  traditional  aethod,  the  results  from  only  one 
run  at  the  lou  value  (U#l)  and  one  run  et  the  high  value  (HNN)  are 
used  for  assessment.  For  the  factorial  method,  the  average  of  four 
values  is  used  to  assess  the  output  effect  at  the  low  value  (Ul, 
UN,  LHl  and  LHH),  For  the  high  value,  the  average  of  IftJL,  HHL,  HLH 
and  MW,  is  used  for  assessment.  Thus,  it  can  be  seen  that  four 
times  as  mjch  data  Is  used  for  the  estimation  of  the  values  at  the 
entrants  (the  hlgh/lou  values)  in  the  factorial  method  as  In  the 
traditional  method.  Move  data  points  are  also  used  In  estimating 
the  hifh/low  values  for  factors  B and  C.  In  addition  to  more  data 
being  need  in  the  estimation  of  the  high/lou  values,  the  interaction 
effects  am  be  estimated  ter  the  AB*  AC  and  8C  interactions,  through 
use  of  the  factorial  method. 
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APPLICATION  OF  STATISTICAL  TECHNIQUES  TO  MODEL  SENSITIVITY  TESTING 


CHAPTER  III 

FRACTIONAL  FACTORIAL  DESIGNS 


1 . Discussion.  - As  the  nunber  of  factors  to  be  considered  In  « 
factorial  experiment  Increases,  the  number  of  factor  level  combina- 
tions Increases  very  rapidly.  For  example,  when  there  are  five 
factors  at  two  levels  each,  the  number  of  combinations  are  2’  * 32 
computer  runs  for  one  replication.  Recently,  for  A Tactical. 
Logistical,  and  Air  Simulation  (ATLAS)  computer  model  (a  theater- 
level  ground  combat  simulation),  it  ms  desired  to  Investigate  10 
factors,  which  would  have  required  2K  * 1024  computer  runs  for 
one  replication.  It  Is  obvious  that  it  would  not  be  economically 
or  timewise  feasib'e  to  make  1024  coogNiter  runs  for  a sensitivity 
experiment.  A methci  that  is  feasible  i;  to  use  a design  called 
fractional  replication  or  fractional  factorial.  Mien  only  a frac- 
tion of  a full  design  is  run,  the  design  is  referred  to  as  a frac- 
tional factorial  design.  The  amount  of  fractionating  that  is  per- 
formed depends  on  the  nunber  of  factors  under  investigation,  the 
number  of  levels  of  interest,  the  order  of  the  interactions  of 
Interest  and  the  total  number  of  computer  runs  that  it  Is  feasible 
to  perform.  For  this  paper,  consider  only  two  levels,  « low  level 
and  a high  level.  Therefore,  only  1/2P  (p  * 1,  2,  3....)  fractional 
replicates,  t.e.,  1/2,  1/4  or  1/8.  etc.,  of  the  full  design,  are 
employed. 

2.  Examples 

a.  For  the  ATLAS  sensitivity  experiment,  in  which  there  were 
10  input  factors  of  interest,  it  was  determined  that  less  than  ISO 
runs  could  be  made.  After  some  research  and  study.  It  was  found 
that  a 1/8  fractional  replicate  of  the  210  experiment  would  take 
128  runs  and  would  provide  the  necessary  information  an  the  10 
main  effects  and  the  45  two-factor  interaction  effects.  £ty  using 
a fractional  factorial  design,  7/8  of  the  runs  (896)  w era  elimi- 
nated 


b.  For  another  theater-level  combat  simulation  model,  the 
Conceptual  Oeslgn  for  the  Amy  in  the  Field  Evaluation  Model  (CEN), 
available  time  and  money  limited  the  number  of  runs  to  16.  There 
were  five  primary  factors  of  Interest.  A full  factorial  design 
would  have  required  2s  * 32  runs.  Through  use  of  a 1/2  fractional 
replicate,  the  nunber  of  runs  was  reduced  to  16  and  It  was  still 
possible  to  estimate  the  5 main  effects  and  the  10  two-factor 
interactions  effects. 
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3.  Caution.  * It  !$  emphasized  that  only  a particular  1/2  of 
the  nms  in  the  2s  design  or  a particular  1/8  of  the  runs  In  the 
2'°  design  ulll  glee  the  desired  results.  The  nroper  runs  are 
determined  by  choosing  the  defining  contrasts  such  that  none  of 
the  factors  of  interest  are  confounded  with  each  other.  These  «/ 
methods  are  explained  in  experimental  design  homes,  e.g.,  by  Davies15 , 
Ksnpthonm&r  and  Hidtss.  There  Is  also  a publication  on  Fractional 
Factorial  Designs  published  by  the  Rational  Bureau  of  Standardly 
that  gives  various  designs  for  factors  at  two  levels. 
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APPLICATION  OF  STATISTICAL  TECHNIQUES  TO  MODEL  SENSITIVITY  TESTING 

CHAPTER  IV 

MINIMIZING  CHANGES  TO  FACTOR  LEVELS 
1.  General  Pi scission 


a.  In  designing  a field  experiment  or  a laboratory  experiment, 
randomization  of  the  order  of  experimentation  is  of  utmost  importance 
owing  to  the  fact  that  certain  variables  cannot  be  controlled. 
Randomization  of  the  order  of  experimentation  will  tend  to  average 
out  the  effect  of  the  uncontrollable  variables  and  eliminate  any 
bias  In  the  estimation  of  the  effects  of  interest.  In  coRfNJter 
models,  randomization  of  the  order  of  runs  is  not  necessary  since 
the  same  answer  will  be  obtained  regardless  of  order  (assuming  the 
same  random  number  seed  is  used).  Since  It  is  not  necessary  to 


randomize  the  order  in  which  conputer  runs  are  made,  it  is  advan- 
tageous to  minimize  the  nunfcer  of  factor  level  changes  and  also  to 
minimize  the  number  of  changes  to  the  factor  levels  that  are  most 
difficult  to  change.  Let's  look  at  a T factorial  design  with  the 
runs  arranged  in  conventional  Yates  order  (See  Figure  IV-1).  In 
run  ramber  one,  we  see  that  all  four  factors  are  at  the  low  level. 

Then  in  run  number  two,  factor  A has  been  changed  to  the  high  level, 
and  the  other  three  factors  remain  unchanged,  in  run  number  three, 
factor  A has  been  changed  back  to  the  low  level  and  factor  B has 
been  changed  to  the  high  level . Counting  down  each  column  the 
number  of  changes  f.xxn  Low  (L)  to  High  (H)  or  vice  versa,  it  can  be 
seen  that  15  factor  level  changes  occur  in  factor  A,  with  7,  3 and 
1 changes  for  factors  B.  C and  D,  respectively,  for  a total  of  26 
factor  level  changes.  The  expected  number  of  factor  level  changes  . 
for  a 2k  factorial  design  In  a completely  randomized  order  is  (k)  2*"*« 
For  a Z4  design,  the  expected  number  of  changes  is  equal  to  32.  The 
minimum  nudber  of  factor  level  changes  that  is  required  for  a 2* 
factorial  design  is  2 - 1 . For  a 2q  factorial  design,  the  minimum 
number  of  changes  required  is  15.  Thus,  there  is  quite  a difference 
In  the  expected  rubber  of  changes  and  the  mini  nun  number  of  changes. 
There  are  over  200  different  run  orders  that  require  only  IS  factor 
level  changes  for  a 2*.  Let's  look  at  one  of  these  (See  Figure  IV-2). 
Notice  that  for  any  two  consecutive  runs  there  is  only  a one  factor 
l«v«l  change.  For  example,  factor  A is  changed  from  the  low  level 
in  run  number  one  to  the  high  level  in  rum  rubber  two  with  the  otter 
three  factors  renaining  at  the  low  level.  In  run  rubber  three, 
factor  D is  changed  and  the  other  three  factors  remain  at  the  same 
level  as  In  number  two.  Moving  down  each  of  the  columns,  note 
that  six,  two,  four  and  three  changes  are  required  for  factors  A,  B, 

C and  D,  respectively.  If  one  factor  requires  more  time  or  effort 
to  change  the  parameters  from  low  to  high  than  the  other  factors. 


iv-l 


the  columns  can  he  rotated  or  Interchanred  without  affecting  the 
design  or  the  total  mater  of  changes.  For  example,  assune  factor  A 
Is  the  aost  difficult  to  change  from  la*  to  high.  Currently.  It  Is 
scheduled  for  the  nost  changes.  However,  by  interchanging  the 
col  wans  under  A and  B.  factor  A now  has  only  two  changes.  This 
technique  can  be  very  useful  when  the  facto*  under  study  Is  an  entire 
array  of  meters  that  is  being  changed  by  a certain  aneent. 

b.  The  technique  discussed,  along  with  an  alaorfthm  for 
generating  these  designs  for  elninue  changes  Is  described  In  the 
February,  1974  issue  of  Technonetri cs  12* 
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FIORE  Ilf-1.  A 2*  Factorial  Design  with  has  Arranged 
In  Conventional  Tates  Order. 
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APPLICATION  OF  STATISTICAL  TECHNIQUES  TO  MODEL  SENSITIVITY  TESTING 


CHAPTER  V 

ANALYSES  OF  OUTPUT  OATA 


1.  Ssasal.  * IT  the  nodel  of  interest  Is  a probablfstlc  model 
and  the  resources,  such  as  tlee  and  money.  permit,  the  experiment 
can  he  repeated  to  perform  an  analysis  of  variance  (AMNIA)  on  the 
data.  The  hypothesis  of  no  change  In  output  values  tdien  the  low 
level  of  a factor  Is  changed  to  the  high  level  of  the  factor  cm 
be  statistically  tested  at  a stated  level  of  confidence.  However, 
for  a deterministic  model,  where  repeating  the  experiment  will 
give  the  same  results,  a within  model  variation  does  met  exist. 
Therefore,  the  factors  cannot  be  statistically  tested.  The  follow- 
leg  paragraphs  describe  some  ways  of  analyzing  the  results  from  a 
deterministic  model.  Prababllstlc  model  results  cm  also  be 
analyzed  In  this  meaner.  In  addition  to  the  ANOKA,  or  In  pitot  of 
the  ANNA  if  the  eaxperieent  cannot  be  repeated. 


2-  fas!* 


a.  Main  Effects.  - For  a look  at  the  analysis,  use  the  Z3 
design  with  some  rfctl t »al  data  (See  Figure  V-T).  To  determine  the 


effect  of  factor  A 
data  for  factor  A at 


the  NOE  output,  the  average  of  the  output 
the  low  level  Is  Obtained  as  follows: 


I - (LLL  ♦ LLH  ♦ LHL  ♦ LHH)/4 
* (K>  ♦ 10  ♦ IS  ♦ 5)/4  « 10 


Similarly  for  the  high  level  of  A: 

F • (HLL  ♦ HUi  ♦ MW.  ♦ WH)/4 
H 

■ (20  ♦ 20  ♦ 25  ♦ 15)/4  * 20 

The  calculation  of  the  B and  C effects  are  dene  In  a similar 

F « (ILL  ♦ UH  ♦ HLL  ♦ HUI}/4 
L 

• (10  ♦ 10  ♦ 20  ♦ ?0>/4  « IS 
(LHL  ♦ LHH  ♦ NHL  ♦ HW0/4 

• (15  ♦ 5 ♦ 25  ♦ 15)/4  « 15 


V-l 


Measure  of  effectiveness  Factor  A 


(UL  ♦ LHL  ♦ HU  ♦ HHL)/4 
■ (10  ♦ 15  ♦ 20  ♦ 25)/4  = 17.5 
(ULH  ♦ LHH  ♦ HLH  ♦ HHH)/4 
« (10  ♦ 5 ♦ 20  ♦ 15)/4  * 12.5 


Figure  1-2  is  * graphical  Illustration  of  the  A,  B and  C effects. 
Factor  C 
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FIORE  ¥-2.  Graphical  Illustration  of  A,  B and  C Effects 
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Thus,  changing  fro*  the  lew  level  to  the  high  level  caused  a sharp 
Increase  In  the  NOE  value  attributed  to  A.  no  change  attributed  to 
B and  a Moderate  decrease  attributed  to  factor  C.  (The  lines  In 
Figure  W are  nerely  for  connecting  associated  points.)  By 
subtracting  the  low  level  from  the  high  level  a warical  nature 
can  be  obtained  far  canperisoe.  For  the  three  factors  this  mature 
would  be: 

X-  20  - 10  - 10 

F-T*  15-15*0 

H L 

V V ,2#5  ‘ 17,5  * 

h-  IflSCCeftlW  Sffyft  - The  di  interaction  tern  aem  Is 
calculated  as follows: 

XT-  (ILL  ♦ LLH)/2  « (M)  ♦ 10)/2  - 10 

XTN  (INI  ♦ li*l)/2  • (15  ♦ 5)/2  « 10 

XT  • (MU  ♦ HLH)/2  » (20  ♦ 20)  ft  = 20 
H L 

XT  - (NHL  ♦ WO/2  - (25  ♦ 15)/2  * 20 

19  91 

Tht  change  in  factor  B when  factor  A is  at  the  low  level  is 
(J^‘  1^)  ■ (10-10)  * 0,  and  the  change  In  factor  B when  factor  A 
Is  at  the  high  level  is  l^-  ]^)  = (20-20)  * 0.  The  AC  inter- 
action tern  wean  1$  calculated  as  fellows: 

KjTJ*  (ILL  «■  LML)/2  « (10  ♦ 15)/2  * 12.5 

XT  • (tiH  ♦ UBQ/2  * (10  ♦ 5)/2  - 7.5 
L H 

XT  « (HU.  ♦ IKL)/2  « (20  ♦ 25)/2  « 22.5 
H L 

TJJTh-  (MLH  ♦ HHH)/2  * (20  ♦ 15)/2  - 17.5 
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in  factor  C when  factor  A is  at  the  low  level  Is, 

) * 02.5  * 7.5)  » 5.0,  and  the  change  In  factor  C vhen 

factor  A is  at  the  high  level  is,  (XT  - ITT  ) * (22.5  - 17.5)  * 5.0. 

H L H H 

The  BC  interaction  tent  nean  Is  calculated  as  follows: 

FT  * (ILL  ♦ HLL)/2  * (10  + 2 0)/2  * 15 
L t 

FT  » (LLH  ♦ HLH)/2  * (10  ♦ 20J/2  » 15 

L H 

FT  - (LHL  ♦ WL)/2  > 05  ♦ ti)f2  - M 

H L 

FT  * (LHH  ♦ HNH)/2  = (5  * 1S1/2  « W 

H M 

The  change  in  factor  C vdien  factor  B u at  tJ«;  low  level  is, 

(FT  - FT  ) * (15  - 15)  - 0.  wniie  the  dwi  , in  C then 

L L L H 

factor  B is  at  the  hf(*i  level  is  (FT  - FT  )-(<»-  M)  = io. 

HI  H m 

The  three  interaction  tern  effects  can  be  qrartovaliy  ii  i os t rated 
as  In  Figure  V-3.  (The  lines  in  Figure  V-3  are  nenply  ft  it  connect- 
ing associated  points.)  Since  the  lines  are  parr  lie  i,  it  can  he 
seen  that  there  Is  absolutely  i<»  interaction  t<*u*er.  fn  t .-rs  A and 
B or  factors  A and  C.  Ifwi  factor  B is  at  Uk  low  level , there  is 
no  change  in  the  Measure  of  Effectiveness  value  as  factor  C is 
changed  from  the  low  level  to  the  high  level . However,  when 
factor  B Is  at  the  high  level,  there  is  a decrease  in  the  Measure 
of  Effectiveness  value  as  factor  C is  Changed  fron  the  law  level 
to  the  hidi  level.  Thus,  one  factor  (factor  C)  behaves  differently 
wider  different  levels  of  the  other  factor  (factor  B).  As  there 
is  an  Interaction  effect  between  factors  B and  C,  -ay  further  test- 
ing involving  either  factor  B or  factor  C will  have  to  include  the 
other  factor  since  factors  B and  C are  not  ind»  indent  of  each  other. 
It  should  be  noted  that  the  results  are  seldom,  if  ever,  as  clear 
cut  as  the  demonstration  example,  rhe  lines  awe  seldom  exactly 
parallel  as  was  demonstrated,  however,  the  degree  of  non-parallelism 
can  be  noted  and  rank  ordered  in  those  cases  that  re  not  clear  cut. 

c.  Reiult.  - Fran  the  «?vanple  oiven,  it  un  tv*  concluded  that 
the  Kjasiwe  cfF  Effectiveness  value  is  most  -ensiti  ••  to  factor  A, 
with  factor  C being  the  next  nost  sensitive,  and  factor  6 being 
hprtant  due  to  the  fact  that  if  interacts  with  f*cto»  ' 
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APPLICATION  OF  STATISTICAL  TECHNIQUES  TO  MODEL  SENSITIVITY  TESTING 


CHAPTER  V! 
CONCLUSION 


1.  Advantages  of  Experimental  Design.  - It  is  hoped  that  the 
dnnonstration  shows  nan  a statistical  tool,  nanely  the  design  of 
experiments,  can  be  used  to  obtain  more  and  better  infbmation  froe 
a sensitivity  analysis  of  a model  than  the  old  traditional  One- 
Factor-At-A-T  ine  approach.  In  very  snail  node  Is  dene  it  can  be 
assuned  that  all  the  factors  are  independent  and  that  interactions 
will  not  occur,  One-Factor-At-A-T ine  sensitivity  txsttnq  would  be 
acceptable.  However,  in  the  large  models  which  «cre  built  to 
represent  complex  processes  involving  naqy  interrelated  parts  or 
ccnponants  which  nay  have  an  Interacting  effect  iron  outowt  variables, 
One-Factor-At-A-T he  analysis  is  not  the  proper  tool.  It  is  believed 
that  use  of  the  de$1*jn  of  experiments  techniques  lead*,  to  a better 
understanding  of  model  sensitivity  In  specific  factors  and  combina- 
tions of  factors . 
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